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METHOD AND DEVICE FOR THE DIGITAL IMAGE PROCESSING OF CMOS CAMERA 
IMAGES 

CROSS REFERENCE TO RELATED APPLICATIONS 
This application is the US national phase of PCT 
s application PCT/DE2003/003939, filed 28 November 2003, published 1 

July 2004 as WO 2004/055727, and claiming the priority of German 
patent application 10258662.4 itself filed 13 December 2002, whose 
entire disclosures are herewith incorporated by reference. 

FIELD OF THE INVENTION 
10 The invention relates to a method as well as to a 

device for the digital image processing of CMOS camera images. 

BACKGROUND OF THE INVENTION 
To be able to process images with a computer system they 
must be converted to a data format which is computer compatible. 
15 This conversion is called digitalizing in digital image processing. 

The original image data is transformed into a computer-conforming 
data format. The transformations can be available as two- 
dimensional or multidimensional functions for the processing. 
Upon the taking of the picture, a continuous scene is spatially 
20 discretized. One possible mathematical description of digitalizing 

image data uses a notation in the form of image matrices . The 
image S (the scene S) is a rectangular matrix (image matrix) S = 
(s(x, y) ) with image rows and image columns. The row index is x 
and the column index is y. The image point (pixel) at a location 
25 (row, column) = (x, y) determines the gray value s(x, y) . Thus 

elemental regions of the scene are each imaged as a pixel of the 
image matrix. For digitalizing the image data, a rastering (grid, 
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scanning) and quantizing are required. In the rastering, the image 
to be digitalized is subdivided into area segments of the raster by 
superimposing on the image a rectangular or square grid. In the 
quantizing each area segment of the raster is assigned a gray value 
s(x, y) of a gray scale G. The determination of this gray value 
can be effected point by point or by averaging over the raster 
area. 

To acquire digital image data, apart from CCD cameras, in 
many cases CMOS cameras are also used. These cameras are most 
commonly employed in science and industry, for example to take 
pictures of crashes and for monitoring rapid technological 
procedures in production. CMOS (Complementary Metal Oxide 
Semiconductor) cameras have, by comparison to CCD image sensors, a 
higher brightness dynamic as well as higher permissible operating 
temperatures . The light particle (photon) impinging on the 
photodiodes of the CMOS camera are converted into electric 
currents. The light-sensitive photodiodes are associated with a 
plurality of transistors. The CMOS camera pixels determine their 
gray values (signal) from the actual photo current of the 
photodiodes. Each pixel clan be individually read and evaluated. 
This enables optional access to the respective image part of 
interest and has special advantages in industrial image processing. 
With the aid of CMOS cameras, very high image rates can be produced 
(extreme time magnification) . The access time for individual 
pixels are thus naturally very short, that is the actual photo 
current has available only a very short time constant. 

At high contrast and with objects that are moving or 
objects which are rapidly changing, at high image rates the not yet 
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decayed strong currents from prior bright image regions tend to 
dominate the later darker signal regions. The "parasitic" 
capacities which are contained in the pixel circuit (see [4] (T. 
Seiffert, Measurement Processes and Parameters for Estimating the 
Dynamic Contrast Resolution Properties of Electronic Cameras, 
Diploma Dissertation, Kahlsruhe University (TH) , 2001) , heading 
8.3, give rise to a blurring in time of the pixel signal. This 
effect is in the context of the present invention, referred to as a 
capacitive afterglow effect. The very high gray value resolutions 
of the camera which result from the usual logarithmic treatment is 
thus greatly reduced or highly error-prone values are supplied. If 
a bright signal, for example, moves over a relatively dark 
background, for example a weld point over a sheet metal workpiece, 
a tail is formed behind it (compare [4] , page 37) . This tail 
overlies the dark background. For example in the welding of sheet 
metal, one must be able to monitor the weld seam directly behind 
the weld point and thus it is necessary to wait until the tail has 
disappeared from the seamed region. 

There are many possibilities for correction of this 
detrimental effect of capacitive afterglow upon the speed of the 
image acquisition: 

By a moderate background lighting, the discharge of 
bright pixels can be significantly accelerated. If the direction 
of movement is known, with certain applications it is possible to 
detect the position of real objects or of projected patterns which 
can be analyzed with respect to the movement direction, that is one 
can detect the rapid change from dark to bright. These processes 
however are suitable only for certain applications when for example 
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lighting or the overall image acquisition situation is known or 
controllable. The indicated strategy for evaluation cannot 
reconstruct nonvisible signal components but rather will only 
ignore or bypass them. 

The evaluation of optical processes, for example, like 
the decay of the temperature in thermal images is carried out by 
means of mathematical models. Thus, for example, from [3] (Horst 
W. Haussecker and David J. Fleet, Computing Optical Flow with 
Physical Models of Brightness Variation IEEE Trans. PAMI , Vol. 23, 
No. 6 pp 661-673, June 2001), FIG. 9, the use of a differential 
equation (DGL) is known. In these processes the physical 
characteristics of the image acquisition is described in a 
differential equation and unknown parameters of this differential 
equation are numerically approximated. Known local approximation 
methods include, among others "ordinary least squares (OLS)", 
"total least squares (TLS) " and "mixed OLS-TLS" [5] ©. Garbe, 
Measuring Heat Exchange Processes at the Air-Water Interface from 
Thermographic Image Sequence Analysis, Doctorate Dissertation, 
Heidelberg University, 2001) , all of which are special forms of the 
least square method and can be drawn from any standard work dealing 
with numerological methods. Furthermore, so-called variation 
methods with data and smoothing procedures are used (see for 
example [1] : B. Jahne, Digital Image Processing, 4 th volume, 
Springer, 1997, and [6]: (J. Weickert and C. Schnorr, Variational 
Optic Flow Computation with a Spatio-Temporal Smoothness 
Constraint, Technical Report 15/2000 Computer Science Series, July 
2000) . 



23314 SN 10/538702 AY 2622 



Clean Copy 



The use of corresponding methods for the evaluation of 
CMOS camera images has not been previously described in the 
literature. 

OBJECT OF THE INVENTION 

It is thus an object of the invention to provide a method 
and device by means of which an improved digital image processing 
of CMOS camera images is possible. 

SUMMARY OF THE INVENTION 

This object according to the invention is attained in 
that the variation in time of the output signal value g is a 
combination of the term c*g and the source term q and the 
calculation of the target signal value q comprises the subtraction 
of the term c*g from the variation in time of the output signal 
value g of the image data, c being a decay constant. 

With the method and device of the invention, it is 
possible immediately to obtain a rapid and direct evaluation of 
moving images with high contrast and without the capacitive after- 
glow effects which have led to a delayed or prolonged evaluation. 
It is possible to evaluate the actual photo current without 
capacitive afterglow effects which cannot be avoided for physical 
reasons. It is also possible to reconstruct high dynamic changes 
from bright to dark (higher contrast) for example in high-speed 
image acquisition. It is possible to evaluate intensities with 
minimum influence of afterglow in a quantitative manner. 

BRIEF DESCRIPTION OF THE DRAWING 

The invention is described as shown in the sole figure to 
the attached drawing and by way of example below: 
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The relationship between capacities, photo currents and 
image data can be given for moving objects with the aid of a 
partial linear differential equation (1) in gray values which can 
be formulated as follows : 

fkiWVJi , ... .... 

di 

+ c{x>yj)g(x>y\t) - q(x,yj) = 0 (1) 

3c 3, 

In this equation: 

g = the gray value of the image sequence 
c = the decay constant (normally negative) 
q = source term of interest (light) 
u = local (object) shift u = (dx/dt, dy/dt) . 
The gray value g of the image sequence represents the 
uncorrected image data directly supplied by the CMOS camera. It is 
a singular output signal. The expression "output signal value" is 
to be understood as meaning, hereinafter, the entire measurement 
signal g produced by the photon current. 

The decay constant c is the physical parameter which 
corresponds to the discharge velocity of the capacitances in the 
CMOS camera, that is the discharge speed of the camera circuit 
capacity. The quantity of charge which has flowed away from the 
camera capacity between readout processes is given by the 
expression c*g. 
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The source term q is the charge at a point in time t 
resulting from the photo current g. Thus q is the target signal of 
interest. 

By the subtraction of the term c*g from the variation 
with time of the output signal value g, the actual photo current 
without the capacitive afterglow effect, namely, the source term q 
can be determined, that source term q being referred to hereinafter 
as the target signal value. 

An advantageous feature of the invention is characterized 
in that a parameter estimation or approximation is carried out. 
This method step is especially suitable for image data with high 
contrast since here the afterglow effects are relevant. These 
regions can be rapidly and simply detected, for example by means of 
the local contrast measurements described in [1] : B. Jahne, Digital 
Processing, 4th Volume, Springer, 1997, by a threshold 
segmentation. If one limits the parameter estimation to these 
regions, a significant acceleration of the computation can be 
achieved . 

From the literature reference [3] (Horst W. Haussecker 
and David J. Fleet, Computing Optical Flow with Physical Models of 
Brightness Variation IEEE Trans. PAMI , Vol. 23, No. 6, pp 661-673, 
June 2201) , the "total least squares (TLS) " method of general 
parameter estimation in partial differential equations of this type 
is known. The method is characterized in that, for example, it 
enables this parameter estimation to be carried out. However, 
other estimation methods are also suitable, especially those partly 
described above and known parameter estimation processes like for 
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example the "ordinary least squares (OLS) " or the "mixed OLS-TLS" 
and/or variability methods . 

In an advantageous feature of the method, the source term 
q is obtained with the aid of a parameter estimation technique like 
for example the method of "total least squares" [3] Horst W. 
Haussecker and David J. Fleet, Computing Optical Flow with Physical 
Models of Brightness Variation IEEE Trans. PAMI , Vol. 23, No. 6, pp 
661-673, June 2001, from the ordinary output signal value. For 
this purpose the above given differential equation (1) , is used. 
From the starting signal g, by means of discrete convolutions with 
derivative kernels, the space-time derivatives of g are determined. 
These and all previously known parameters , for example from 
calibration steps or image accession conditions , are fed into the 
equation (1) . From the thus resulting system of equations, q (x, 
y, t) is obtained by parameter estimation. 

An advantageous refinement of the method according to the 
invention is characterized in that the decay constant c can also be 
determined by parameter estimation or approximation. 

An advantageous refinement of the method according to the 
invention is characterized in that the decay constant c can be 
determined by a prior calibration of the CMOS camera. In that case 
it may be determined by parameter estimation. As a consequence, an 
acceleration of the process is enabled. The decay constant c is 
under sufficiently constant operating conditions, only a magnitude 
which does not vary with time and is dependent upon the camera, 
that is c = c (x, y) . With good homogeneity of the pixel, c = 
const. The decay constant c can thus be calculated also off-line 
in a calibration step. 
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An advantageous refinement of the method according to the 
invention is characterized in that the object movement u can also 
be determined by parameter estimation. The already known object 
movements need not be approximated. The parameters u x and u y can 
then be introduced into the differential equation (1) . The 
parameter approximation is thereby simplified and accelerated. 

An advantageous refinement of the method is characterized 
in that the method according to the invention is implemented with 
the aid of FPGA' s (Field Programmable Gate Arrays) . If one limits 
the output signal value to image segments which are not too large, 
it is then possible to achieve image rates of several images per 
second. 

The invention relates further to a device for digital 
image processing in CMOS cameras which is characterized in that it 
is suitable to carry out the method according to the invention. 
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